APPARATUS AND METHOD FOR IMAGE CODING AND DECODING 
CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] The present application claims priority from 

Japanese Application Nos . P2000-165298 filed June 2, 2000 and 
P2001-001031 filed January 9, 2001, the disclosures of which 
are hereby incorporated by reference herein. 
BACKGROUND OF THE INVENTION 

[0002] The present invention relates generally to an image 

coding apparatus and method, an image decoding apparatus and 
method, and a recording medium. More specifically, the present 
invention relates to an image coding apparatus and method, an 
image decoding apparatus and method, and a recording medium 
which are suitably for use in apparatus for re-encoding video 
streams and recording and reproducing the re-encoded video 
streams . 

[0003] Digital television broadcasts such as European DVB 

(Digital Video Broadcast) , American DTV (Digital Television) 
broadcast, and Japanese BS (Broadcast Satellite) digital 
broadcast use MPEG (Motion Picture Expert Group) 2 transport 
streams. A transport stream consists of continuous transport 
packets, each packet carrying video data or audio data, for 
example. The data length of one transport packet is 18 8 bytes. 
[0004] Unlike analog television broadcasts, digital 

television broadcasts are capable of providing services added 
with multimedia coding data. In these services, data such as 
video data, audio data, character graphics data, and still 
picture data, for example, are associated with each other for 
transmission by the multimedia coding data. For the multimedia 
coding data, a coding method based on XML (Extensible Markup 
Language) is used in the Japanese BS digital broadcast, for 
example. The details of this method is disclosed in ARIB STD- 
B24 Data Coding And Transmission Specification for Digital 
Broadcasting, for example. 

[0005] The data such as video data, audio data, character 

graphics data, and still picture data are each packetized into 
a transport packet for transmission. 

[0006] FIGS. 1A and IB show an example of synthesizing 

data to be transferred between the sending and receiving sides 
and a multimedia screen. As shown in FIG. 1A, the sending side 
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sends to the receiving side video data, character graphics 
data for displaying buttons A through C, text data for 
displaying "XYZABC...," and multimedia coding data for 
relating these data to each other. The sending side generally 
denotes a television broadcast station, for example. However, 
it herein denotes a television broadcast station which 
includes a recording apparatus (the recording side) which 
receives and records data transmitted from broadcast stations, 
the example illustrated in FIG. 1A including the data which is 
output from this recording apparatus. 

[0007] The multimedia coding data includes data which can 

synthesize on the receiving side video data, character 
graphics data, and text data and display the synthesized data. 
To be more specific, the multimedia coding data includes the 
data associated with the display positions of the video, 
character graphics, and text which are displayed by the size- 
associated data such as the multimedia plane (the display area 
of images on the television receiver, for example) size 
(plane_height and plane_width) and video display size 
(video_height and video_width) , video data, character graphics 
data, and text data, as shown in FIG. IB. 

[0008] On the basis of the multimedia coding data, the 

receiving side processes the video data, the character 
graphics data, and the text data to display a resultant image, 
as shown in FIG. IB. 

[0009] Through the screen on which the above-mentioned 

image is displayed, the user can receive services such as 
displaying desired information in the video section by 
clicking button A corresponding to that information and 
obtaining, from the text data displayed in the bottom of the 
screen, the information associated with the matter displayed 
in the video section, for example. 

[0010] If a television program carried by a transport 

stream transmitted from a digital television broadcast is 
recorded without change to a recording medium on the received 
side, the program can be recorded without its picture and 
audio qualities being deteriorated at all. However, in order 
to record as long a television program as possible to a 
recording medium having a limited recording capacity by 
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presupposing a certain degree of picture quality deterioration, 
the received video stream must be decoded and then encoded 
again to lower the bit rate of the transport stream. 
[0011] For example, the re-encoding of the video stream of 

a television program attached with multimedia coding data to 
lower its bit rate for recording may be implemented by sub- 
sampling the image to change writing blocks. However, this 
approach presents a problem of causing a mismatch in the 
relationship between the video stream resulting from re- 
encoding and the multimedia coding data. The following 
describes an example of this mismatch with reference to FIGS. 
2A and 2B. 

[0012] In the example shown in FIG. 2A, the sending side 

(the recording side) converts the original video writing block 
to a smaller picture frame at the time of re— encoding . 
Therefore, as shown in FIG. 2B, on the receiving side (the 
reproducing side), changes occur in the video display size and 
position, resulting in a display screen which is different 
from the display screen intended by the sending side (the 
display screen to be displayed on the basis of the data before 
being re-encoded) . 
SUMMARY OF THE INVENTION 

[0013] It is therefore an object of the present invention 

to provide an image coding apparatus and method, an image 
decoding apparatus and method, and a recording medium, wherein 
information that no mismatch is caused in the relationship 
between a video stream after re-encoding and other data is 
generated and recorded on the recording side, and wherein it 
is prevented a shrunk picture frame, for example, from 
occurring on the reproducing side by referencing the 
information generated on the recording side. 

[0014] In carrying out the invention and according to a 

first aspect thereof, there is provided a first image coding 
apparatus comprising: inputting means for inputting a 
multiplexed stream containing multimedia coding data; 
separating means for separating a video stream from the 
multiplexed stream input by the inputting means; converting 
means for performing a predetermined converting process on the 
video stream separated by the separating means; generating 
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means for generating additional information indicating that a 
mismatch will occur when the converted video stream is 
displayed on the basis of the multimedia coding data; and 
outputting means for outputting the converted video stream, 
the multimedia coding data, and the additional information. 

[0015] The first image coding apparatus may further 

comprise: coding means for coding the additional information 
generated by the generating means as data separate from the 
multiplexed stream containing the converted video stream. 

[0016] The first image coding apparatus may further 

comprise: coding means for multiplexing the additional 
information generated by the generating means with the 
multiplexed stream containing the converted video stream and 
then coding a multiplexed result. 

[0017] The converting means may convert a video stream 

picture frame parameter. 

[0018] The conversion by the converting means may include 

at least a process of decoding the video stream separated by 
the separating means and a process of encoding the decoded 
video stream. 

[0019] The additional information generated by the 

generating means may contain at least one of original picture 
frame information and an original screen aspect ratio. 

[0020] The additional information generated by the 

generating means may contain an original video format and a 
video format after the conversion. 

[0021] The additional information generated by the 

generating means may contain an original screen aspect ratio 
and a screen aspect ratio after the conversion. 

[0022] The additional information generated by the 

generating means may contain at least one of information 
indicating whether or not a picture frame of the video stream 
has been converted by the converting means, information about 
an original picture frame of the video stream separated by the 
separating means, and an original screen aspect ratio. 

[0023] In carrying out the invention and according to a 

second aspect thereof, there is provided a first image coding 
method comprising the steps of: inputting a multiplexed stream 
containing multimedia coding data; separating a video stream 
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from the multiplexed stream input in the inputting step; 
performing a predetermined converting process on the video 
stream separated in the separating step; generating additional 
information indicating that a mismatch will occur when the 
converted video stream is displayed on the basis of the 
multimedia coding data; and outputting the converted video 
stream, the multimedia coding data, and the additional 
information . 

[0024] The first image coding method may further comprise 

the step of: coding the additional information generated in 
the generating step as data separate from the multiplexed 
stream containing the converted video stream. 

[0025] The first image coding method may further comprise: 

a coding step of multiplexing the additional information 
generated in the generating step with the multiplexed stream 
containing the converted video stream and then coding a 
multiplexed result. 

[0026] A video stream picture frame parameter may be 

converted in the converting step. 

[0027] The conversion in the converting step may include 

at least a process of decoding the video stream separated in 
the separating step and a process of encoding the decoded 
video stream. 

[0028] The additional information generated in the 

generating step may contain at least one of original picture 
frame information and an original screen aspect ratio. 

[0029] The additional information generated in the 

generating step may contain an original video format and a 
video format after the conversion. 

[0030] The additional information generated in the 

generating step may contain an original screen aspect ratio 
and a screen aspect ratio after the conversion. 

[0031] The additional information generated in the 

generating step may contain at least one of information 
indicating whether or not a picture frame of the video stream 
has been converted in the converting step, information about 
an original picture frame of the video stream separated in the 
separating step, and an original screen aspect ratio. 
[0032] In carrying out the invention and according to a 
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third aspect thereof, there is provided a first recording 
medium recorded a computer-readable program, the program 
comprising the steps of: inputting a multiplexed stream 
containing multimedia coding data; separating a video stream 
from the multiplexed stream input in the inputting step; 
performing a predetermined converting process on the video 
stream separated in the separating step; generating additional 
information indicating that a mismatch will occur when the 
converted video stream is displayed on the basis of the 
multimedia coding data; and output ting the converted video 
stream, the multimedia coding data, and the additional 
information . 

[0033] In carrying out the invention and according to a 

fourth aspect thereof, there is provided a second recording 
medium recorded a video stream converted by a predetermined 
conversion process, multimedia coding data, and additional 
information indicating occurrence of a mismatch at displaying 
the converted video stream on the basis of the multimedia 
coding data. 

[0034] The additional information may be coded and 

recorded as data different from a multiplexed stream 
containing the converted video stream. 

[0035] The additional information may be coded and 

recorded as multiplexed with a multiplexed stream containing 
the converted video stream. 

[0036] The video stream may be converted in its picture 

frame parameter. 

[0037] The video stream may be decoded and then encoded. 

[0038] The additional information may contain at least one 

of original picture frame information and an original screen 
aspect ratio. 

[0039] The additional information may contain information 

about an original video format and information about a video 
format after the conversion. 

[0040] The additional information may contain information 

about an original screen aspect ratio and information about a 
screen aspect ratio after the conversion. 

[0041] The additional information may contain at least one 

of information indicating whether or not a picture frame of 
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the video stream has been converted, information about an 
original picture frame of the video stream, and information 
about an original screen aspect ratio. 

[0042] In carrying out the invention and according to a 

fifth aspect thereof, there is provided a first image decoding 
apparatus comprising: inputting means for inputting a 
multiplexed stream containing multimedia coding data; 
separating means for separating a video stream from the 
multiplexed stream input by the inputting means; decoding 
means for decoding the video stream separated by the 
separating means; and processing means for performing a 
predetermined conversion process on the decoded video stream 
in accordance with additional information indicating 
occurrence of a mismatch at displaying the decoded video 
stream on the basis of the multimedia decoding data. 
[0043] The first image decoding apparatus may further 

comprise: acquiring means for acquiring the additional 
information from data different from the multiplexed stream. 
[0044] The first image decoding apparatus may further 

comprise: acquiring means for acquiring the additional 
information from a multiplexed stream with which the 
additional information is multiplexed. 

[0045] The processing means may convert a picture frame 

parameter of the video stream. 

[0046] The conversion by the processing means may include 

at least a process of decoding the video stream separated by 
the separating means and a process of encoding the decoded 
video stream. 

[0047] The additional information may contain at least one 

of information about an original picture frame and information 
about an original screen aspect ratio. 

[0048] The additional information may contain an original 

video format and information about a video format after the 
conversion . 

[0049] The additional information may contain information 

about an original screen aspect ratio and information about a 
screen aspect ratio after the conversion. 

[0050] The additional information may contain at least one 

of information indicating whether or not a picture frame of 
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the video stream has been converted by the converting means, 
information about an original picture frame of the video 
stream separated by the separating means, and information 
about an original screen aspect ratio. 

[0051] In carrying out the invention and according to a 

sixth aspect thereof, there is provided a first image decoding 
method comprising the steps of: inputting a multiplexed stream 
containing multimedia coding data; separating a video stream 
from the multiplexed stream input in the inputting step; 
decoding the video stream separated in the separating step; 
and performing a predetermined conversion process on the 
decoded video stream in accordance with additional information 
indicating occurrence of a mismatch at displaying the decoded 
video stream on the basis of the multimedia decoding data. 
[0052] The first image decoding method may further 

comprise the step of: acquiring the additional information 
from data different from the multiplexed stream. 

[0053] The first image decoding method may further 

comprise the step of: acquiring the additional information 
from a multiplexed stream with which the additional 
information is multiplexed. 

[0054] A picture frame parameter of the video stream may 

be converted in the processing step. 

[0055] The conversion in the processing step may include 

at least a process of decoding the video stream separated in 
the separating step and a process of encoding the decoded 
video stream. 

[0056] The additional information may contain at least one 

of information about an original picture frame and information 
about an original screen aspect ratio. 

[0057] The additional information may contain an original 

video format and information about a video format after the 
conversion . 

[0058] The additional information may contain information 

about an original screen aspect ratio and information about a 
screen aspect ratio after the conversion. 

[0059] The additional information may contain at least one 

of information indicating whether or not a picture frame of 
the video stream has been converted in the converting step, 
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information about an original picture frame of the video 
stream separated in the separating step, and information about 
an original screen aspect ratio. 

[0060] In carrying out the invention and according to a 

seventh aspect thereof, there is provided a third recording 
medium recorded a computer-readable program, the program 
comprising the steps of: inputting a multiplexed stream 
containing multimedia coding data; separating a video stream 
from the multiplexed stream input in the inputting step; 
decoding the video stream separated in the separating step; 
and performing a predetermined conversion process on the 
decoded video stream in accordance with additional information 
indicating occurrence of a mismatch at displaying the decoded 
video stream on the basis of the multimedia decoding data. 
[0061] In carrying out the invention and according to an 

eighth aspect thereof, there is provided a second image coding 
apparatus comprising: inputting means for inputting a 
multiplexed stream; separating means for separating a video 
stream from the multiplexed stream input by the inputting 
means; determining means for determining whether or not 
multimedia coding data is contained in the multiplexed stream 
input by the inputting means; generating means, if it is 
determined by the determining means that the multimedia coding 
data is contained in the multiplexed stream, for generating 
coding control information for instructing not to change a 
display format of the video stream separated by the separating 
means; converting means for performing a predetermined 
conversion process on the video stream separated by the 
separating means on the basis of the coding control 
information generated by the generating means; and 
multiplexing means for generating a multiplexed stream which 
contains the video stream converted by the converting means. 
[0062] The generating means may instruct not to change any 

of a picture frame, a video format, and an aspect ratio. 
[0063] In carrying out the invention and according to a 

ninth aspect thereof, there is provided a second image coding 
method comprising the steps of: inputting a multiplexed 
stream; separating a video stream from the multiplexed stream 
input in the inputting step; determining whether or not 
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multimedia coding data is contained in the multiplexed stream 
input in the inputting step; generating coding control 
information for instructing not to change a display format of 
the video stream separated in the separating step, if it is 
determined in the determining step that the multimedia coding 
data is contained in the multiplexed stream; performing a 
predetermined conversion process on the video stream separated 
in the separating step on the basis of the coding control 
information generated in the generating step; and generating a 
multiplexed stream which contains the video stream converted 
in the converting step. 

[0064] Coding control information for instructing not to 

change any of a picture frame, a video format, and an aspect 
ratio may be generated in the generating step. 

[0065] In carrying out the invention and according to a 

tenth aspect thereof, there is provided a fourth recording 
medium recorded a computer-readable program, the program 
comprising the steps of: inputting a multiplexed stream; 
separating a video stream from the multiplexed stream input in 
the inputting step; determining whether or not multimedia 
coding data is contained in the multiplexed stream input in 
the inputting step; generating coding control information for 
instructing not to change a display format of the video stream 
separated in the separating step, if it is determined in the 
determining step that the multimedia coding data is contained 
in the multiplexed stream; performing a predetermined 
conversion process on the video stream separated in the 
separating step on the basis of the coding control information 
generated in the generating step; and generating a multiplexed 
stream which contains the video stream converted in the 
converting step. 

[0066] In carrying out the invention and according to an 

eleventh aspect thereof, there is provided a fifth recording 
medium recorded coding control information instructing not to 
change a display format of a video stream and a multiplexed 
stream containing a video stream on which a predetermined 
conversion process has been performed on the basis of the 
coding control information. 

[0067] As described and according to the first image 
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coding apparatus and method and the program stored in the 
first recording medium, a video steam is separated from a 
multiplexed stream containing multimedia coding data, and a 
predetermined conversion process is performed on the separated 
video stream, and additional information indicating that a 
mismatch will occur when displaying the converted video stream 
on the basis of the multimedia coding data. 

[0068] The second recording medium stores a video stream 

converted by a predetermined conversion process, the 
multimedia coding data, and the additional information 
indicating that a mismatch will occur when displaying the 
converted video stream on the basis of the above-mentioned 
multimedia coding data. 

[0069] As described and according to the image decoding 

apparatus and method and the program stored in the third 
recording medium, when a video stream is separated from an 
input multiplexed stream, the separated video stream is 
decoded, and the decoded video stream is displayed on the 
basis of multimedia coding data, a mismatch occurs. On the 
basis of the additional information about this mismatch 
occurrence, a predetermined conversion process is performed on 
the decoded video stream. This novel configuration prevents 
the mismatch from occurring between the video stream and the 
multimedia coding data. 

[0070] As described and according to the second image 

coding apparatus and method and the program stored in the 
fourth recording medium, a video stream is separated from an 
input multiplexed stream, the input multiplexed stream is 
checked whether or not multimedia coding data is contained and, 
if the multimedia coding data is contained, coding control 
information for giving an instruction not to change the 
display format of the separated video stream is generated, and 
a predetermined conversion process is performed on the 
separated video stream on the basis of the generated coding 
control information. 

[0071] The fifth recording medium also stores the above- 

mentioned coding control information giving instruction not to 
change the display format of a video stream and a multiplexed 
stream containing the video stream on which a predetermined 
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conversion process has been performed on the basis of the 

coding control information. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0072] These and other objects of the invention will be 

seen by reference to the description, taken in connection with 
the accompanying drawing, in which: 

[0073] FIGS. 1A and IB are schematic diagrams illustrating 

a display screen to be shown on the basis of multimedia coding 
information; 

[0074] FIGS. 2A and 2B are schematic diagrams illustrating 

a mismatch which takes place when a video stream is re- 
encoded ; 

[0075] FIG. 3 is a block diagram illustrating a recording 

apparatus practiced as one embodiment of the present 
invention; 

[0076] FIGS. 4A and 4B illustrate the operation of a 

multiplexer shown in FIG. 3; 

[0077] FIGS. 5A, 5B and 5C illustrate the processing by an 

arrival timestamp adding block; 

[0078] FIG. 6 illustrates multimedia display sub- 

information; 

[0079] FIG. 7 illustrates an example of Programlnf o ( ) 

syntax; 

[0080] FIG. 8 illustrates an example of StreamCodinglnf o ( ) 

syntax ; 

[0081] FIG. 9 illustrates the meaning of s tream_coding 

type; 

[0082] FIG. 10 illustrates the meaning of video_f ormat ; 

[0083] FIG. 11 illustrates the meaning of frame_rate; 

[0084] FIG. 12 illustrates the meaning of 

display_aspect_ratio; 

[0085] FIG. 13 is a flowchart describing the processing of 

coding AV stream and multimedia display sub-information; 
[0086] FIG. 14 is a flowchart describing the coding 

processing to be executed for restricting the re-encoding of a 
multiplexed stream video including multimedia coding data; 
[0087] FIG. 15 illustrates an example of an input 

transport stream; 

[0088] FIG. 16 illustrates an example of a transport 
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stream after the re-encoding of the video stream shown in FIG. 
15; 

[0089] FIG. 17 is a flowchart describing a recording rate 

control process by a recording apparatus shown in FIG. 3; 
[0090] FIG. 18 is a flowchart describing another recording 

rate control process by the recording apparatus shown in FIG. 
3; 

[0091] FIG. 19 illustrates another example of a transport 

stream resulting from the re-encoding of the video stream; 
[0092] FIG. 20 illustrates another example of the input 

transport stream; 

[0093] FIG. 21 is a block diagram illustrating a 

configuration of a reproducing apparatus practiced as one 
embodiment of the present invention; 

[0094] FIGS. 22A and 22B illustrate a display screen to be 

shown when multimedia display sub-information is added; 
[0095] FIG. 23 is a block diagram illustrating another 

configuration of the recording apparatus practiced as one 
embodiment of the present invention; 

[0096] FIG. 24 is a flowchart describing the processing of 

reproducing an AV stream which uses multimedia display sub- 
information; 

[0097] FIG. 25 is a block diagram illustrating another 

configuration of the reproducing apparatus practiced as one 

embodiment of the present invention; and 

[0098] FIG. 26 illustrates recording media. 

DETAILED DESCRIPTION 

[0099] This invention will be described in further detail 

by way of example with reference to the accompanying drawings . 
Now, referring to FIG. 3, there is shown a block diagram 
illustrating an exemplary configuration of a recording 
apparatus 1 practiced as one embodiment of the invention. A 
transport stream received at an antenna, not shown, is input 
in a selector 10. A program number (a channel number) 
specified by the user is also input from a terminal 11 to the 
selector 10. Referring to the received program number, the 
selector 10 extracts the specified program from the received 
transport stream and outputs a partial transport stream. The 
partial transport stream is input in a demultiplexer 12 and an 
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analyzing block 13. 

[0100] The partial transport stream input in the 

demultiplexer 12 is separated into a video stream and other 
streams (audio, still picture, character graphics, and 
multimedia coding data for example) . The video stream thus 
obtained is output to a decoder 14. The other streams are 
output to a multiplexer 16. In addition to the transport 
packets other than video, the demultiplexer 12 outputs the 
output timing information in the input transport stream of 
these transport packets to the multiplexer 16. 

[0101] The decoder 14 applies a predetermined decoding 

scheme, for example, MPEG2 to the input video stream and 
outputs the decoded video data to an encoder 15. Also, the 
decoder 14 outputs the stream information about the video 
stream obtained at decoding to a coding controller 18. 

[0102] On the other hand, the analyzing block 13 analyzes 

the input transport stream to obtain the stream information 
about the non-video streams, for example, a bit rate and 
outputs it to the coding controller 18. The stream information 
about the non-video streams output from the analyzing block 13, 
the video stream information output from decoder 14, and a 
stream recording bit rate output from a terminal 19 are input 
in the coding controller 18. From these data, the coding 
controller 18 sets the video data coding conditions (coding 
control information) to be executed by the encoder 15 and 
outputs these coding conditions to the encoder 15 and a coding 
block 20. 

[0103] The coding controller 18 uses, as a bit rate to be 

allocated to the video data encoding, a value obtained by 
subtracting a total value (the data input from the analyzing 
block 13) of the bit rates of the non-video streams from a 
stream recording bit rate (the data input, via the terminal 19, 
from a controller, not shown, for controlling the operation of 
the recording apparatus 1, for example) . The coding controller 
18 sets coding control information such as bit rate and 
picture frame such that an optimum picture quality can be 
achieved with the bit rate thus obtained and outputs this 
coding control information to the encoder 15 and the coding 
block 20. The details of the coding control information will 
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be described later with reference to FIGS. 15 through 20. 

[0104] When a stream is recorded to a recording medium 

with a fixed rate, this stream recording bit rate becomes the 
fixed rate; if a stream is recorded with a variable bit rate, 
this stream recording bit rate is a mean bit rate per 
predetermined time. However, the maximum value of the variable 
bit rate in this case needs to be lower than the maximum 
recording bit rate ensured by the recording medium concerned. 

[0105] The encoder 15 encodes (on the basis of MPEG2 for 

example) the video data output from the decoder 14 on the 
basis of the coding control information output from the coding 
controller 18 and outputs the resultant video data to the 
multiplexer 16. The video stream from the encoder 15, the 
transport stream packets other than video from the 
demultiplexer 12, and the information about the occurrence 
timing of the transport stream packets other than video are 
input in the multiplexer 16. On the basis of the input 
occurrence timing information, the multiplexer 16 multiplexes 
the video stream with the transport stream packets other than 
video and outputs the result to the arrival timestamp adding 
block 17 as a transport stream. 

[0106] FIGS. 4A and 4B schematically illustrate the above- 

mentioned processing to be executed by the multiplexer 16. FIG 
4A shows the timing of the input transport stream packets. In 
these figures, the cross-hatched portions indicate the video 
packets while the white portions indicate the stream packets 
other than video. As shown in FIG. 4A, the input transport 
stream packets are continuous; however, the data volume of the 
video data is reduced by the re-encoding of video data by the 
encoder 15. Consequently, the number of video packets is 
reduced . 

[0107] As shown in FIG. 4B, the multiplexer 16 does not 

change the timing of the stream packets other than video but 
causes only the timing of the video packets to be different 
from the original state (shown in FIG. 4A) . 

[0108] As shown in FIGS. 5A, 5B and 5C, the arrival 

timestamp adding block 17 adds a header (TP_extra_header ) 
including an arrival timestamp to each of the packets (FIG. 
5A) of the input transport stream to generate a source packet 
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(FIG. 5B) , arranges the generated source packets continuously 
(FIG. 5C) , and outputs them to a writing block 21. The arrival 
timestamp is information indicative of the timing with which 
the transport stream packets occur in a transport stream. The 
writing block 21 puts the input source packet stream 
consisting of continuous source packets and records the file 
to a recording medium 22. It should be noted that the 
recording medium 22 may be any type of recording medium. 
[0109] The information output from the coding block 20 is 

also input in the writing block 21. On the basis of the video 
coding information from the coding controller 18, the coding 
block 20 generates multimedia display sub-information and 
outputs the same to the writing block 21. The multimedia 
display sub-information to be output to the writing block 21 
is information for keeping the video display position and 
display size unchanged on multimedia plane from those of the 
image (the image which would be displayed without re-encoding) 
intended by the sending side even if the picture frame size 
has changed by transcoding (decoding by the decoder 14 and 
then encoding by the encoder 15) a video stream. This 
information is also used at time of reproduction in 
combination with multimedia coding data. 

[0110] The following describes the multimedia display sub- 

information more specifically. As shown in FIG. 6, the 
multimedia display sub-information consists of three flags of 
a mismatch flag (mismatch_MMinf o_f lag ) , a re-encoded flag 
(Re_encoded_f lag) , and a frame size change flag 
(changed_f rame_size_f lag) , data associated with two sizes 
indicative of an original horizontal size 

( original_horizontal_size ) and an original vertical size 
( original_vertical_size ) , and an original screen aspect ratio 
(original_display_aspect_ratio) . 

[0111] The mismatch flag indicates whether there exists a 

mismatch in the relationship between video and multimedia 
coding data. The re-encoded flag indicates whether the video 
has been re-encoded at the time of recording. The frame size 
change flag indicates whether the picture frame of video has 
been changed by re-encoding for example . The original 
horizontal size indicates the horizontal size of a picture 
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frame before re-encoding. The original vertical size indicates 
the vertical size of a picture frame before re-encoding. The 
original screen aspect ratio indicates the aspect ratio of a 
frame screen before re-encoding. 

[0112] It should be noted that the above-mentioned 

multimedia display sub-information is illustrative only. 
Therefore, information other than shown in FIG. 6 may be 
included in or part of the information shown in FIG. 6 may be 
excluded from the multimedia display sub-information. 
[0113] The following describes another example of the 

multimedia display sub-information. In the following example, 
the multimedia display sub-information is stored in a 
Programlnf o ( ) syntax shown in FIG. 7. The following describes 
the fields associated with the present invention in the 
Programlnf o ( ) syntax. 

[0114] "length" indicates the number of bytes between the 

byte just after the length field and the last byte of 
Programlnf o ( ) inclusive. 

[0115] "num_of_program_sequences" indicates the number of 

program sequences in the an AV stream file. A source packet 
sequence with which the program contents specified by this 
format in the AV stream file are constant is referred to as a 
program sequence. 

[0116] "SPM_program_sequence_start" indicates an address 

at which the program sequence starts in the AV stream file. 
"SPM_program_sequence_start" is of a size in unit of source 
packet number and counted from the initial value 0 starting 
with the first packet of the AV stream file. 

[0117] "program_map_PID" is value of the PID of a 

transport packet having PMT (Program Map Table) applicable to 
that program sequence. 

[0118] "num_of_streams_in_ps" indicates the number of 

elementary streams defined in that program sequence. 

[0119] "stream_PID" indicates the value of the PID for the 

elementary stream defined in the PMT which is referenced by 
the program map PID of that program sequence. 

[0120] "StreamCodinglnf o ( ) " indicates the information 

about the elementary stream indicated by the above-mentioned 
stream PID. 
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[0121] FIG , 8 shows the syntax of StreamCodinglnf o ( ) . 

"length" indicates the number of bytes between the byte just 
after this length field and the last byte of 
StreamCodinglnf o ( ) inclusive. 

[0122] "stream_coding_type" indicates the coding type of 

the elementary stream indicated by the stream PID for this 
StreamCodinglnf o () . The meanings of the individual types are 
shown in FIG. 9. 

[0123] If the value of stream coding type is 0x02, it 

indicates that the elementary stream indicated by the stream 
PID is a video stream. 

[0124] If the value of stream coding type is OxOA, OxOB, 

or OxOD, it indicates that the elementary stream indicated by 
the stream PID is multimedia coding data. 

[0125] If the value of stream coding type is 0x06, it 

indicates that the elementary stream indicated by the stream 
PID is subtitles or teletext. 

[0126] "video_f ormat" indicates the video format of a 

video stream indicated by the stream PID for this 
StreamCodinglnf o () . The meanings of the individual video 
formats are shown in FIG. 10. 

[0127] In FIG. 10, 480i indicates video display of NTSC 

standard TV (interlace frame of 720 pixels x 480 lines) . 576i 
indicates video display of PAL standard TV (interlace frame of 
720 pixels x 576 lines) . 480p indicates video display of 
progressive frame of 720 pixels x 480 lines. 1080i indicates 
video display of interlace frame of 1920 pixels x 1080 lines. 
720p indicates video display of progressive frame of 1230 
pixels x 720 lines. 

[0128] "frame_rate" indicates the frame rate of a video 

stream indicated by the stream PID for this StreamCodinglnf o ( ) 
The meanings of the individual frame rates are shown in FIG. 
11. 

[0129] "display_aspct_ratio" indicates the display aspect 

ratio of a video stream indicated by the stream PID for this 
StreamCodinglndo ( ) . The meaning of the individual aspect 
ratios are shown in FIG. 12. 

[0130] "original_video_f ormat_f lag" indicates whether 

there exist original video format and original display aspect 
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ratio in this StreamCodinglnf o ( ) . 

[0131] "original_video_f ormat" indicates a video format 

before a video stream indicated by the stream PID for this 
StreamCodinglnf o ( ) is coded. The meanings of the individual 
original video formats are the same as shown in FIG. 10. 
[0132] "original_display_aspect_ratio" is the display 

aspect ratio before a video stream indicated by the stream PID 
for this StreamCodinglnf o ( ) is coded. The meanings of the 
individual aspect ratios are the same as shown in FIG. 12. 
[0133] It is assumed that, in transcoding a transport 

stream with a multimedia data stream (BML stream or subtitles) 
multiplexed along with a video stream, the re-encoding of the 
video stream changes its video format (for example, from 1080i 
to 480i), while the multimedia data stream retain its original 
stream contents. In this case, a mismatch in information may 
occur between a new video stream and the multimedia data 
stream. For example, although the parameters associated with 
the display of the multimedia data stream are determined on 
the supposition of the video format of the original video 
stream, the video format may be changed by the re-encoding of 
the video stream. 

[0134] The video format of the original video stream is 

indicated by the video format and the display aspect ratio. 
The video format of the re-encoded video stream is indicated 
by the original video format and the original display aspect 
ratio . 

[0135] If there exists a mismatch between the values of 

the video format and the original video format and/or between 
the display aspect ratio and the original display aspect ratio, 
it indicates that a video format change has been caused by the 
video re-encoding at the time of recording. 

[0136] If the stream PID in which the stream coding type 

indicates multimedia coding data and subtitles is included in 
Programlnf o ( ) , it indicates that the multimedia data is 
multiplexed in an AV stream file (a transport stream) . 

[0137] If Programlnf o ( ) indicates that a video format 

change has been caused by the re-encoding of video at the time 
of recording and multimedia data is multiplexed in the AV 
stream file, then it is determined that there exists a 
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mismatch in display between the video stream (re-encoded) and 
the multimedia data (the original multimedia data) in the AV 
stream file. 

[0138] In such a case, the information about the original 

video stream, namely the original video format and the 
original display aspect ratio, becomes effective. The 
reproducing apparatus generates a display screen from the 
above-mentioned new video stream and multimedia data stream as 
follows . 

[0139] The video stream is up-sampled to a video format 

indicated by the original video format and the original 
display aspect ratio. 

[0140] The up-sampled image and the multimedia data stream 

are synthesized to form a correct display screen. 
[0141] The multimedia display sub-information generated by 

the coding block 20 is recorded by the writing block 21 to the 
recording medium 22 but stored as a file which is different 
from the source packet stream file output from the arrival 
timestamp adding block 17. If the multimedia display sub- 
information is recorded by the writing block 21 to the 
recording medium 22 as a file different from the source packet 
stream file, the filed multimedia display sub-information is 
output from the coding block 20. 

[0142] FIG. 13 is a flowchart describing the processing of 

coding an AV stream and multimedia display sub-information. 

[0143] In step 50, a multiplexed stream including 

multimedia coding data is input in the recording apparatus 1. 

[0144] In step 51, the demultiplexer 12 separates the 

video stream from the multiplexed stream. 

[0145] In step 52, the encoder 15 re-encodes the video 

stream decoded by the decoder 14. 

[0146] In step 53, the multiplexer 16 multiplexes the 

above-mentioned video stream and multimedia coding data to 
generate a multiplexed stream. 

[0147] In step 54, the coding block 20 generates 

multimedia display sub-information. 

[0148] In the above description, the coding controller 18 

generates the coding control information including bit rate 
and picture frame on the basis of the input data. The coding 
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controller 18 may generate the following information as 
alternative coding control information. Namely, if the input 
transport stream is found including multimedia coding data by 
the analyzing block 13, then the coding controller 18 may 
generate coding control information, when encoding is executed 
by the encoder 15, for instructing the encoder 15 to execute 
the re-encoding with a picture frame (the picture frame before 
re-encoding) of the same size as that of the picture frame of 
the original video and output the generated coding control 
information to the encoder 15. 

[0149] When the above-mentioned method is used, the 

encoder 15 re-encodes the video data supplied from the decoder 
14 with the same value as that of the picture frame of the 
original video stream on the basis of the input coding control 
information. If such coding control information is generated 
and the re-encoding is executed on the basis of the coding 
control information, no picture frame change is caused by the 
re-encoding, thereby preventing a mismatch from occurring in 
the relationship between the video stream obtained by re- 
encoding and the multimedia coding data. 

[0150] Still alternatively, the following information may 

be generated as the coding control information generated by 
the coding controller 18. Namely, if the input transport 
stream is found including multimedia coding data by the 
analyzing block 13, then the coding controller 18 may generate 
coding control information, when encoding is executed by the 
encoder 15, for instructing the encoder 15 to execute the re- 
encoding under the same conditions as the video format (shown 
in FIG. 10) and screen aspect ratio (shown in FIG. 12) of the 
original video and output the coding control information to 
the encoder 15. 

[0151] When the above-mentioned method is used, the 

encoder 15 re-encodes the video supplied from the decoder 14 
under the same conditions as the video format (shown in FIG. 
10) and screen aspect ratio (shown in FIG. 12) of the original 
video on the basis of the input coding control information. If 
such coding control information is generated and the re- 
encoding is executed on the basis of the coding control 
information, no video format and no screen aspect ratio change 
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is caused by the re-encoding, thereby preventing a mismatch 
from occurring in the relationship between the video stream 
obtained by re-encoding and the multimedia coding data. 
[0152] FIG. 14 is a flowchart describing the coding for 

restricting the re-encoding of the video of a multiplexed 
stream including multimedia coding data. 

[0153] In step 70, a multiplexed stream is input in the 

recording apparatus 1. 

[0154] In step 71, the demultiplexer 12 separates the 

video stream from the multiplexed stream. 

[0155] In step 72, the analyzing block 13 checks if the 

multimedia coding data is included in the video stream. If the 
multimedia coding data is included, the analyzing block 13 
sends the coding control information to the encoder 15 
instructing the same to re-encode the video stream without 
changing the display format. On the basis of the supplied 
control information, the encoder 15 re-encodes the video 
stream. 

[0156] In step 73, the multiplexer 16 generates a 

multiplexed stream including the above-mentioned video stream. 

[0157] The following describes one example of control to 

be executed on the basis of the coding control information 
with reference to FIGS. 15 through 20. 

[0158] It is assumed here that a transport stream to be 

input in the selector 10 have a constant bit rate R T as shown 
in FIG. 15 for example. The video stream and the non-video 
streams are coded by variable bit rates. In the example shown 
in FIG. 15, in unit time (for example, GOP) A, the bit rate of 
the video stream is R V a and the bit rate of non-video streams 
is R 0A - In unit time B, the bit rate of the video stream is R VB 
and the bit rate of non-video streams is R OB - In unit time C, 
the bit rate of the video stream is R V c and the bit rate of 
non-video streams is R 0 c- 

[0159] If the transport stream as shown in FIG. 15 is re- 

encoded to output the transport stream having fixed bit rate S 
(S < Ri) as shown in FIG. 16 from the multiplexer 16, the 
coding controller 18 executes the processing described by the 
flowchart shown in FIG. 17. 

[0160] First, in step SI, the coding controller 18 sets to 
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S the bit rate (recording rate) of a transport stream to be 
output from the multiplexer 16 on the basis of a control 
signal input from a controller, not shown, via the terminal 19. 
Next, in step S2, the coding controller 18 determines non- 
video streams to be recorded and computes a maximum total 
value D of the bit rates of the determined streams . 
[0161] The maximum value D is determined from the stream 

specification of the input transport stream. For example, if 
two audio streams are to be recorded in addition to the video 
stream, the maximum value D is 384 x 2 Kbps since the maximum 
value of the bit rate of one audio stream is 384 Kbps 
according to the Japanese digital BS broadcast stream 
specification . 

[0162] In step S3, the coding controller 18 uses value C 

obtained by subtracting the maximum value D computed in step 
S2 from the recording bit rates set in step SI (= S - D) , as a 
bit rate to be allocated to the re-encoding of the video data. 
In step S4, the coding controller 18 analyzes the coding 
information such as the video stream bit rate and picture 
frame from the video stream information output from the 
decoder 14 . 

[0163] In step S5, the coding controller 18 determines, on 

the basis of the value C computed in step S3 and the video 
stream coding information analyzed in step S4, a video coding 
parameter (video coding control information) such that an 
optimum picture quality is achieved. 

[0164] For example, in the example shown in FIG. 16, value 

S is 1/2 of value Ri- In the present example, the bit rate of 
steams other than video is the maximum value D, which is used 
without change as the bit rate of non-video steams in a 
multiplexed stream after re-encoding. 

[0165] Then, video coding parameters are determined such 

that an optimum picture quality can be achieved within the 
range of (S - D) . If the picture frame is controlled, the 
horizontal direction of a picture frame of 720 x 480 pixels 
for example is sampled by 1/2 into 360 x 480 pixels. The 
determined coding parameters (bit rate and picture angle) are 
supplied to the encoder 15 as video coding control information. 
[0166] In step S6, on the basis of the video coding 
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control information supplied from the coding controller 18, 
the encoder 15 re-encodes the video data of unit time (in this 
example, unit time A) to be processed now. In the example 
shown in FIG. 16, the actual bit rate R 0 a is smaller than the 
maximum value D in unit time A; however, since the maximum 
value D is fixed, the video allocated bit rate becomes (S - D) . 
A wasted portion Rsa which cannot be used for video coding 
occurs because the maximum value D is fixed, the wasted 
portion being filled with stuffing bits. 

[0167] In step S7, the coding controller 18 determines 

whether there remains any stream to be re-encoded. If there 
still remains any stream to be re-encoded, the procedure 
returns to step S4 to repeat the above-mentioned processes. 
[0168] If, in step S7, there remains no more stream to be 

re-encoded, this processing comes to an end. 

[0169] Thus, in the example shown in FIG. 16, the bit rate 

of non-video streams is D also in unit time B and the video 
stream allocated bit rate is S - D because it is fixed. 
Stuffing bits are inserted in value R sb (= S - (S - D) - R 0B = D 
- Rob) • 

[0170] In unit time C, too, the bit rate of non-video 

streams is D and the video stream allocated bit rate is S - D. 
It should be noted that, in unit time C, 

[0171] D = R oc , so that there exists no stuffing bits. 

[0172] Thus, in the example shown in FIG. 16, the video 

stream is coded with a fixed bit rate . 

[0173] FIG. 18 is a flowchart describing a processing 

example in which the video re-encoding allocated bit rate is 
variable. First, in step S21, the coding controller 18 sets 
recording rate S on the basis of the information supplied via 
the terminal 19. Next, in step S22, the coding controller 18 
analyzes the coding information of the video stream on the 
basis of the video stream information supplied from the 
decoder 14. The processes of steps S21 and S22 are the same as 
those of steps SI and S4 of FIG. 17. 

[0174] In step S23, the coding controller 18 computes, 

from the output of the analyzing block 13, the total bit rate 
B in each unit time of non-video streams . 

[0175] In step S24, the coding controller 18 uses, as the 
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video re-encoding allocated bit rate, value C (= S - B) 
obtained by subtracting value B obtained in step S23 from 
value S obtained in SI. 

[0176] In step S25, the coding controller 18 determines, 

on the basis of value C obtained in step S24 and a result of 
analysis of the video stream coding information obtained in 
step S22, video coding parameters such that an optimum picture 
quality is obtained. The determined coding parameters are 
output to the encoder 15. 

[0177] In step S26, the encoder 15 re-encodes the video 

data of the current unit time on the basis of the coding 
parameters determined in step S25. Consequently, as shown in 
FIG. 19 for example, after allocation of R oa (=Roa) as the bit 
rate in unit time of non-video streams, the bit rate of the 
video stream is set to bit rate R va specified by (S - R oa ) - 
[0178] In step S27, the coding controller 18 determines 

whether there remains any stream to be processed. If there 
still remains any stream to be processed, the procedure 
returns to step S22 to repeat the above-mentioned processes. 
If there remains no more stream to be processed, this 
processing comes to an end. 

[0179] Thus, in unit time B, after allocation of bit rate 

Rob (= S - R 0 b) of non-video streams, the remaining R vb (= S - 
R ob ) is the bit rate of the video stream. In unit time C, the 
bit rate of the video stream is set to R vc (= S - R 0 c) except 

for bit rate Roc of non-video streams . 

[0180] Thus, in the present processing example, the bit 

rate of the video stream is variable and therefore no stuffing 
bit is needed or the number of stuffing bits can be reduced, 
thereby coding the video stream more efficiently. 
[0181] In the above, the input transport stream has a 

fixed bit rate. The present invention is also applicable to an 
example in which the bit rate of the input transport stream is 
variable as shown in FIG. 20. 

[0182] Consequently, a transport stream of longer content 

can be recorded to the recording medium 22 at a lower bit rate 
as required. 

[0183] In addition, the above-mentioned novel constitution 

prevents the qualities of audio data, still picture and 
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character graphics data, multimedia coding data, and other 
non-video data from being conspicuously deteriorated. These 
non-video data is basically smaller in data volume than video 
data, so that reducing their bit rate in the same ratio as the 
bit rate of video data makes the effects on these non-video 
data relatively greater than those on video data. The novel 
constitution can prevents these effects from being caused. 
[0184] The following describes the reproduction of a 

source packet stream file recorded on the recording medium 22. 
Referring to FIG. 21, there is shown a block diagram 
illustrating the configuration of a reproducing apparatus 
practiced as one embodiment of the invention. A source packet 
stream file recorded on the recording medium 22 is read by a 
reading block 31. The reading block 31 also reads multimedia 
display sub-information recorded on the recording medium 22 as 
a file separate from the source packet stream file. 
[0185] The source packet stream read by the reading block 

31 is output to a arrival timestamp separating block 32 and 
the multimedia display sub-information to a synthesizing block 
36. The arrival timestamp separating block 32 incorporates a 
reference clock. The arrival time stamp separating block 32 
compares the reference clock with the value of the arrival 
timestamp added to the source packet of the input source 
packet stream and, when a match is found, removes the arrival 
timestamp from the source packet having the matching arrival 
timestamp, outputting the resultant packet to a demultiplexer 
33 as a transport stream packet. 

[0186] The demultiplexer 33 separates the input transport 

stream into a video/audio stream and data streams such as 
multimedia coding data, character graphics, text, and still 
picture. Of these separated data, the video/audio stream is 
output to an AV decoder 34, the multimedia coding data to the 
synthesizing block 36, and the data stream such as character 
graphics, text, and still picture to a character 
graphics/still picture decoder 35. 

[0187] The AV decoder 34 separates the input video/audio 

stream into video data and audio data, decodes each data, and 
outputs the decoded audio data to an audio reproducing device, 
not shown, and the decoded video data to the synthesizing 
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block 36. The character graphics/still picture decoder 35 
decodes the input data stream such as character graphics, text, 
and still picture and outputs the decoded character graphics 
data, text data, and still picture data to the synthesizing 
block 36. 

[0188] In the synthesizing block 36, the video data from 

the AV decoder 34, the multimedia coding data from the 
demultiplexer 33, the data from the character graphics /still 
picture decoder 35, and the multimedia display sub-information 
from the reading block 31 are input. Checking the mismatch 
flag (FIG. 6) of the input multimedia display sub-information, 
the synthesizing block 36 determines whether there exits a 
mismatch in the relationship between the input video signal 
and the multimedia coding data. 

[0189] If there exists a mismatch between the value of 

video format and the value of original video format shown in 
FIG. 8 and/or a mismatch between the value of display aspect 
ratio and the original display aspect ratio, the synthesizing 
block 3 6 determines that a video format change has been caused 
by the video re-encoding at the time of recording, detecting a 
mismatch in the relationship between the input video signal 
and the multimedia encoding data. If there exists no mismatch 
between the value of video format and the value of original 
video format and no mismatch between the value of display 
aspect radio and the value of original display aspect ratio, 
the synthesizing block 36 determines that there exists no 
mismatch in the relationship between the input video signal 
and the multimedia coding data. 

[0190] If a mismatch is found in the relationship between 

the input video signal and the multimedia coding data, the 
synthesizing block 36 further references the original 
horizontal size and vertical size of the multimedia display 
sub-information or references the original video format and 
the original display aspect ratio. Then, the synthesizing 
block 36 scale-converts the input video signal so that it can 
be displayed in a frame of the referenced size. On the basis 
of the multimedia coding data, the synthesizing block 36 
outputs the video signal with the scale-converted video signal 
and the data such as character graphics synthesized on a 
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multimedia plane to a television receiver, not shown, which 
servers as a display device. 

[0191] On the other hand, if no mismatch is found in the 

relationship between the input video signal and the multimedia 
coding data, the synthesizing block 36 synthesizes the input 
video signal with other data on a multimedia plane without 
scale conversion and outputs the synthesized data. 
[0192] Thus, recording the multimedia display sub- 

information and using it at the time of reproduction allow the 
receiving side to display a screen as intended on the sending 
side. Referring to FIG. 22, if the re-encoding on the sending 
side (recording side) results in a smaller video picture frame 
than the original, the size reduction is recorded as 
multimedia display sub-information, which is referenced at the 
time of reproduction. Consequently, because there exists no 
mismatch between video data and other data, the receiving side 
(the reproduction side) can display the same screen as the 
original . 

[0193] FIG. 24 is a flowchart describing AV stream 

reproduction processing which uses multimedia display sub- 
information . 

[0194] In step 60, a multiplexed stream including 

multimedia coding data is read from a recording medium and 
input in a reproduction device. 

[0195] In step 61, multimedia display sub-information is 

input. This information is read from the recording medium in 
the case of the reproducing device shown in FIG. 21; in the 
case of a reproducing device shown in FIG. 25, this 
information is separated from the multiplexed stream. 
[0196] In step 62, a video stream is separated from the 

multiplexed stream. 

[0197] In step 63, the video stream is decoded. 

[0198] In step S64, if there exists a mismatch is display 

between the video data and the multimedia coding data, the 
synthesizing block 36 scale-converts the video data on the 
basis of the multimedia display sub-information. 

[0199] In step 65, the synthesizing block 36 synthesizes 

the processed image and the multimedia data to generate a 
display image. 
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[0200] As described, the multimedia display sub- 

information may be recorded on the recording medium 2 as a 
file which is different from the source packet stream file 
containing character graphics data and video signals. 
Alternatively, the mutlimedia display sub-information may be 
embedded in a source packet stream file and then recorded on 
the recording medium 22. FIG. 23 shows the configuration of 
the recording apparatus 1 in which the multimedia display sub- 
information is embedded in a source packet stream file. 
[0201] In comparison between the configuration of the 

recording apparatus 1 shown in FIG. 23 and the configuration 
shown in FIG. 3, the former outputs the multimedia display 
sub-information output from the coding block 2 0 is supplied to 
the multiplexer 16. The multiplexer 16 then generates a 
transport packet of the input multimedia display sub- 
information embeds it into a source packet stream file, 
outputting the same to the arrival timestamp adding block 17 . 
Instead of embedding the multimedia display sub-information 
into a source packet stream file as a transport packet, the 
multimedia display sub-information may be written to a user 
data area in an MPEG video stream. 

[0202] In the present embodiment of the invention, video 

data may be re-encoded in other methods than that described 
above; for example, an input video stream may be converted in 
the DCT area to covert the coding parameters such as picture 
frame . 

[0203] FIG. 25 shows the configuration of the reproducing 

apparatus 30 in which the multimedia display sub-information 
is embedded in a source packet stream file to be recorded on 
the recording medium 22. In comparison between the 

configuration of the reproducing apparatus shown in FIG. 25 
and the configuration shown in FIG. 21, the former reads 
through the reading block 31 only the source packet stream. 
The source packet stream read by the reading block 31 is input 
in the demultiplexer 33 via the arrival timestamp separating 
block 32. 

[0204] The demultiplexer 33 extracts the multimedia 

display sub-information from the input source packet stream 
file and outputs the extracted information to the synthesizing 
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block 36. The further processing is the same as that of the 
configuration shown in FIG. 5. 

[0205] Thus, if the multimedia display sub-information is 

recorded as embedded in a source packet stream file, the 
receiving side can also obtain the video picture size and 
display position intended by the sending side. 

[0206] In the present embodiment of the invention, a 

transport stream was used for example. The present invention 
is also applicable to multiplexed streams such as a program 
stream. 

[0207] The above-described sequence of processing 

operations can be executed by hardware as well as software. In 
the software approach, the recording apparatus 1 (and the 
reproducing apparatus 30) is constituted by a personal 
computer as shown in FIG. 26. 

[0208] Referring to FIG. 26, a CPU (Central Processing 

Unit) 101 executes various processing operations as instructed 
by programs stored in a ROM (Read Only Memory) 102 or loaded 
from a storage block 108 into a RAM (Random Access Memory) 103. 
The RAM 103 also stores, as required, the data necessary for 
the CPU 101 to execute various processing operations. 
[0209] The CPU 101, the ROM 102, and the RAM 103 are 

interconnected via a bus 104. The bus 104 is also connected to 
an input/output interface 105. 

[0210] The input/output interface 105 is connected to an 

input block 10 6 such as a keyboard and a mouse, a display 
device such as CRT or LCD, an output block 107 such as a 
speaker, a storage block 108 such as hard disk, and a 
communication block 109 such as modem or terminal adapter. The 
communication block 109 executes communication processing via 
a network. 

[0211] The input/output interface 105 is also connected to 

a drive 110 as required, in which a magnetic disc 121, an 
optical disc 122, a magneto-optical disc 123, or a 
semiconductor memory 124 is loaded. Computer programs read 
from these storage media are installed in the storage block 
108 as required. 

[0212] The execution of a sequence of processing 

operations by software requires the use of a computer having a 
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dedicated hardware device storing beforehand the programs 
constituting the software or a general-purpose computer in 
which these programs are installed as required from a 
recording medium. 

[0213] The program recording medium for storing computer- 

readable and executable programs may be a package medium which 
is distributed to users providing programs and constituted by 
the magnetic disk 121 or (including floppy disk) , the optical 
disc 122 (including CD-ROM (Compact Disc-Read Only Memory) and 
DVD (Digital Versatile Disc) ) , the magneto-optical disk 123 
(including MD (Mini Disk)), or the semiconductor memory 124 or 
a ROM 102 or a hard disk which is preinstalled in a personal 
computer and provided for users and on which the programs are 
stored temporarily or permanently as shown in FIG. 26. 
[0214] It should be noted that the steps describing the 

programs to be stored in the program storage medium are not 
only executed in a time-dependent manner in the order 
described, but also in parallel or in a discrete manner. 
[0215] As described and according to the first image 

coding apparatus and method and the program stored in the 
first recording medium, a video steam is separated from a 
multiplexed stream containing multimedia coding data, a 
predetermined conversion process is performed on the separated 
video stream, and additional information indicative that a 
mismatch will occur when displaying the converted video stream 
on the basis of the multimedia coding data. 

[0216] The first recording medium stores the converted 

video stream, the multimedia coding data, and the additional 
information indicative that a mismatch will occur when 
displaying the converted video stream on the basis of the 
above-mentioned multimedia coding data. 

[0217] Consequently, in any case, the reproducing side can 

prevent a mismatch from occurring between the video stream and 
the multimedia coding data. 

[0218] As described and according to the image decoding 

apparatus and method and the program stored in the second 
recording medium, when a video stream is separated from an 
input multiplexed stream, the separated video stream is 
decoded, and the decoded video stream is displayed on the 
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basis of multimedia coding information, a mismatch occurs. On 
the basis of the additional information about this mismatch 
occurrence, a predetermined conversion process is performed on 
the decoded video stream. This novel configuration prevents 
the mismatch from occurring between the video stream and the 
multimedia coding data. 

[0219] As described and according to the second image 

coding apparatus and method and the program stored in the 
third recording medium, a video stream is separated from an 
input multiplexed stream, the input multiplexed stream is 
checked for multimedia coding data and, if the multimedia 
coding data is found, coding control information for giving an 
instruction not to change the display format of the separated 
video stream is generated, and a predetermined conversion 
process is performed on the separated video stream on the 
basis of the generated coding control information. 
[0220] The second recording medium also stores the above- 

mentioned coding control information giving instruction not to 
change the display format of a video stream and a multiplexed 
stream containing the video stream on which a predetermined 
conversion process has been performed on the basis of the 
coding control information. 

[0221] Consequently, in any case, the reproduction side 

can prevent a mismatch from occurring between the video stream 
and the multimedia coding data. 

[0222] Although the invention herein has been described 

with reference to particular embodiments, it is to be 
understood that these embodiments are merely illustrative of 
the principles and applications of the present invention. It 
is therefore to be understood that numerous modifications may 
be made to the illustrative embodiments and that other 
arrangements may be devised without departing from the spirit 
and scope of the present invention as defined by the appended 
claims . 
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